Zero-Shot Temporal Action Detection via Vision-Language Prompting

Existing temporal action detection (TAD) methods rely on large training data including segment-level annotations, limited to recognizing previously seen classes alone during inference. Collecting and annotating a set for each class of interest is costly hence unscalable. Zero-shot TAD (ZS-TAD) resolves this obstacle by enabling pre-trained model recognize any unseen classes. Meanwhile, ZS-TAD also much more challenging with significantly less investigation. Inspired the success zero-shot image classification aided vision-language (ViL) models such as CLIP, we aim tackle complex task. An intuitive method integrate an off-the-shelf proposal detector CLIP style classification. However, due sequential localization (e.g., generation) design, it prone error propagation. To overcome problem, in paper propose novel zero- $$\underline{S}$$ hot $$\underline{T}$$ emporal $$\underline{A}$$ ction via Vision- $$\underline{L}$$ anguag $$\underline{E}$$ prompting (STALE). Such design effectively eliminates dependence between breaking route propagation in-between. We further introduce interaction mechanism improved optimization. Extensive experiments standard video benchmarks show that our STALE outperforms state-of-the-art alternatives. Besides, yields superior results supervised over recent strong competitors. The PyTorch implementation available https://github.com/sauradip/STALE .